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This paper considers maximum- likelihood sequence estimation 
(mlse) for quadrature amplitude modulation (qam) signaling at rates 
approaching several baud/Hz. In this regime, intersymbol interfer- 
ence and possibly cross channel coupling are the dominant transmis- 
sion impairments. We derive the structure of a detector that optimally 
accommodates both impairments. A bit error rate performance bound 
is found, and the concept of an error state transition matrix is 
introduced to facilitate the analysis. We explore a modulation scheme 
wherein cross- channel coupling is intentionally introduced, and find 
that it improves detection efficiency. The use of mlse may be an 
important consideration for power and spectrally efficient digital 
radio systems, either terrestrial or satellite, since rates approaching 
the Shannon limit may be attainable without channel coding, and 
frequency selective fading is handled in an optimum manner. 

I. INTRODUCTION 

The search for digital radio modulations which combine power and 
bandwidth efficiency with ease of implementation has attracted inter- 
est for many years. 1 " 9 Combined with the performance analysis of 
receivers which represent a compromise between optimality and prac- 
ticality, this field remains the focus of much current research activ- 
ity. 10-15 Quadrature amplitude modulation (qam) is a particularly sim- 
ple modulation to implement, and determination of the theoretical bit- 
error rate (ber) achievable for this modulation at an arbitrary rate of 
information transfer per unit bandwidth is the subject of this paper. 
Phase shift keying (psk) is, of course, a special case of qam. 

The primary impairment to qam transmission is intersymbol inter- 
ference caused by bandlimiting at the transmitter and dispersion in 
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the channel. We investigate maximum-likelihood sequence estimation 
(mlse) to optimally detect the impaired signal, which is also corrupted 
by additive Gaussian noise. In particular, we analyze the Viterbi 
algorithm for realizing mlse and then proceed to find an upper bound 
for the ber. In this regard, the analysis to be presented simplifies, 
unifies, and expands earlier treatments of the same subject. 16,17 The 
primary goal is to obtain a basis against which the performance of any 
suboptimum receiver may be compared. In so doing, we show qam to 
be capable of providing performance close to the Shannon limit, an 
observation of considerable importance for communication satellite 
and terrestrial digital radio applications. Additionally, study of the 
mechanisms responsible for mlse error generation provides insight 
into appropriate waveshaping to improve ber performance. One such 
waveshaping technique, wherein the spectrum is intentionally asym- 
metric with respect to the transmitting filter passband (e.g., qam- 
Single Sideband), is shown to generally improve the ber performance. 
In Section II, motivation behind a study of qam is illustrated, the 
qam model is presented, and the mlse algorithm is derived. Section 
III is devoted to the derivation of a ber outer bound, and in Section 
IV, this bound is applied to examples which demonstrate the inherent 
power of mlse. We show that a ber of 10" 4 can be maintained at a 
transmission rate of 5 bits/s/Hz with an energy per bit penalty no 
greater than 1 dB compared against ideal nonoverlapping rectangular 
signaling; a four-pole Butterworth transmit filter is assumed in this 
calculation. 

II. QAM MODEL AND THE MLSE ALGORITHM 

Figure la gives a general model of a digital communication system. 
Here, the source produces a stream of binary data d = {do, d\, • • • , 
di } which is converted into an analog waveform by the modulator for 
transmission to the destination. The modulator may include a band- 
limiting filter to contain spectral emission. White Gaussian noise is 
added in the channel, and it is the function of the receiver to reproduce 
a copy cf of the original data sequence with a reasonably low probability 
of bit error. 

The output of the modulator can be represented by the expression 

S G (t) - Re {[A(t, d) +jB(t, d)]e- jUot ] , (1) 

where w is the carrier radian frequency and A(t, d) and B{t, d) are 
data-dependent real waveforms. 

The form of Sdt) provides great flexibility to achieve both high 
power efficiency and high spectral efficiency. However, the modulator 
to generate the best Sdt) and the receiver to detect the data in the 
presence of corrupting noise may be quite difficult to implement, e.g., 
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separate waveform synthesizers and receiver matched filters may be 
required for each of the 2 L possible outcomes of the data stream d. 

To simplify the modulation and detection processes, we consider 
qam, a special subset of (1) having the form 



-jUQt 



where 



and 



S(t) = Re \ £ [a k (d) +jb k (d)]h(t - kT)e 



h(t) = h R {t) +jhi(t) is a complex waveshape, 

a* and bk are real numbers dependent upon the data, 

T" 1 is the channel signaling rate, 



N + 1 is the number of complex channel symbols used to 
transmit the data d. 



(2) 



Equation (2) can readily be modified to admit the familiar staggered 
types of modulation for which the in-phase and quadrature pulses are 
offset from each other by the amount T/2. 6,9 Although mlse can be 
applied to this case, we restrict our attention in the following to 
nonstaggered modulation. We expect that performance similar to what 
we report here is achievable with staggered modulation as well. 

The waveform S(t) may be generated as shown in Fig. lb. Here, the 
numbers a* and bk are generated by a channel encoder and the 
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Fig. 1 — Models for digital communication channels, (a) General digital channel, (b) 
Symbol-by-symbol qam digital channel. 
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numbers a*, bk, a*+y, bk+j can be dependent. In the subsequent devel- 
opment we assume that, far/' 5^ 0, the pairs (a*, 6*) and (a*+>, bk+j) are 
independent, corresponding to independent symbol-by-symbol signal- 
ing. The mlse algorithm to be derived can be generalized to optimally 
detect d when the numbers a* and bk are dependent only upon the 
most recent K data symbols supplied to the encoder. A convolutional 
encoder is one such case which has been treated for real hit) and 
binary signaling. 18 

We note that the numbers a* and bk may be dependent. Thus, our 
treatment admits the familiar psk type of modulation. Further, with 
no loss in generality, we assume in the following that hit) is the 
combined impulse response of the transmitter and a possibly dispersive 
channel. 

At the receiver, we observe 

R(t) = S(t) + Re (DM*) +jn 2 (t)]e- jU0t }, (3) 

where riiit) and /12(f) are independent white Gaussian noise processes, 
each having two-sided spectral density No/2. Thus, the in-phase and 
quadrature components of R{t) are as follows: 

N 

fit) = £ [akhnit - kT) - bkhiit - kT)] + rait) 

4 Suit) + rait) (4) 



and 



^ (*) - 2 [akhiit - kT) + b k hnit - kT)] + n 2 it) 

k=0 

^Siit)+n 2 it). (5) 



Let 



a = {ao, ai, • • • , <in} , b = {60, 61, • • • , bit] . (6) 

Then, since the noise is white and Gaussian, the optimum receiver 
selects as detected data that sequence pair (d, 6) for which the 
likelihood function, 

L(a, b) = exp(- J- J {[f{t) - S R {t)f + [2 it) - S t (t)]*)dt\ , (7) 

is maximized. This is equivalent to maximizing, with respect to the 
hypothesis (a, b), the quantity 



A(a, b) = 2 I [fit)S R {t; a, b) + S. (*)&(*; a, b)]dt 

[S 2 R it;a,b) + SJit;a,b)]dt. (8) 
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From (4) and (5), 

n r- 

A(a, b) = 2 £ J(t)[a k h R (t - kT) - b k h T (t - kT)]dt 
k-o J_ ao 

N f" 

+ 2£ 3(*)[a*A/(*-*T) + &*/**(* -*T)]d* 

- 2 I I [akh R (t - kT) - bkhiit - kT)] 

k-0 m-0 J_ oo 

-[a m h R (t - mT) - b m hi(t - mT)]dt 

A_0 m-0 / „ 



[a m hi(t - mT) + b m h R (t - mT)]dt. (9) 



Let 



a* - | J(t)h R (t - kT)dt, p h - y«)M< " Wcfc, 

2(t)h R (t-kT)dt, Pk = £{t)hi{t-kT)dt. (10) 



Y *= J 2(t)h R (t-kT)dt, Pk = 



Then, the quantities in (10) can be generated by processing both J(t) 
and .2 (t) through filters matched to h R (t) and hi(t), and sampling the 
output at time t = kT. 

Let 

y k = cik + Pa, Zk = yn- fik. (11) 

Finally, let us define 

X*- m = I [M*-*T)M*-TOT)+M*-*r)A/(*-mT)]<ft, (12) 



-j [h R (t- 

-I 



f*-»- [W*-ikDW*-»D-fe(*-*W-»r)]*. (13) 
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Then 


N 


A(a, b) 


= 2 £ (a*^* + bkZk) 

k-0 




N N 

- £ £ («Aa m + bkb m )x k -m 

k-0 m=0 

N N 

~ £ £ ( a A&m — 6*Om) f A 

A— m— 



(14) 

Thus, for each hypothesis under test, the maximum-likelihood re- 
ceiver forms a linear combination of the received statistics from which 
is subtracted a stored constant which is independent of the received 
data. (This constant must be slowly updated if the channel frequency 
characteristics change as during frequency-selective fading. Our anal- 
ysis applies to each instantaneous characteristic since the detection 
epoch is far smaller than the time intervals associated with such 
changing characteristics.) 

We note that the matched filtering operation can be performed in 
the passband, and that the received statistics y* and Zk are, respectively, 
the in-phase and quadrature samples of the passband matched filter 
response to h (t), 

v(t) =^-\ \H(o)\*e*"du = I h(T)K* (t - t)dr. (15) 

Then 

v(nT) = X n+J$n, (16) 

and we see that the quantities \n and f„ are the real and imaginary 
samples of the matched filter response to h(t) at time nT. These are, 
then, real and imaginary components of the intersymbol interference. 
From (12) and (13), 

X~n = Xn, f-»--f», fo-0. (17) 

In the following, we assume that the intersymbol interference vanishes 
for | n | > M, a positive integer. 

For qam, we can use the Viterbi algorithm to maximize (14) with 
respect to (a, b). Rewriting (14), 



N 

A(a, b) = J a k 



k-0 



A-l "I 

2y k - cikXo - 2 X ( a mXk-m + W*-») 



m=k-M 



N 

+ I b k 



A-l 

2z k - bkXo ~ 2 £ (b m x k - m ~ aM-m) I , (18) 



m-k-M 



where the convention is adopted that a* and bk are identically zero for 
k<0. 
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Let the partial sum A„(a, b) be the first n terms in each of the sums 
appearing in eq. (18). Then 



A„(a, b) = A„-i(a, b) + a n 



2v„ - a n xo - 2 



n-l 
m=n—M 
+ b n 2Zn ~ bnXO ~ 2 



n-l 
m=n—M 



(19) 



Equation (19) is in a form such that the Viterbi algorithm can be 
directly applied. We see that to calculate A n (a, b), we must add to 
A„_i (a, b) a term which depends on the received data y n and z n and 
the hypothesis subvectors (a n -M, • • • , a n ) and (b n -M, • • • , b„). Fur- 
thermore, the next calculation to find A„+i(a, b) no longer depends on 
a n -M and b n -M- Thus, we can perform mlse by means of a trellis 
diagram containing I M states, where I is the number of discrete values 
which can be assumed by each symbol pair (a„, 6„). Let us label each 
state by the 2Af-tuple aj-M+i, • • • , a,; bj-M+i, ••• , bj. Then, permissible 
state transitions take the form shown in Fig. 2, drawn for (a„, b n ) 
independent and binary (hence, 1 = 4) and M = 2. For each permissible 
transition, we can compute the branch metric A, (a, b) — A/-i(a, b) 
[see eq. (19)]. 

Maximum-likelihood sequence estimation is performed via recursive 
application of (19) for n - 0, 1, • • • ,j, • • • , N. Suppose the partial sum 
of the most likely path leading into each state for n = j — 1 is known. 
We calculate the partial sum of the / competing paths leading into 
each state for n = j by adding the partial sum of the most likely path 
into each state at n=j — 1 to the branch metric corresponding to that 
transition. The largest of the partial sums among the paths merging at 
each state then corresponds to the most likely path leading to that 
state and is stored for future calculations. The hypothesis {Oj-u, bj-M) 
corresponding to the surviving path at each state is also stored as part 
of the most likely path leading to that state since Oj-M t bj-M does not 
affect subsequent calculations. Thus, at each node, I — 1 out of / 
possible paths are deleted from further consideration. The path 
through the trellis with the greatest metric An then identifies the most 
likely transmitted sequence (a, h). 

Unlike the case of the general digital modulation of (1), the com- 
plexity of the ML receiver for qam grows exponentially with M (the 
one-sided extent of the isi) rather than exponentially with the message 
length. 
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III. BIT-ERROR RATE PERFORMANCE BOUND 

To find an upper bound on the ber performance of mlse, we invoke 
the concept of an error event. Consider an incorrect path which 
diverges from the correct path at depth p in the trellis and remerges 
with it for the first time at depth q. Since the correct and incorrect 
paths agree over the first p epochs, the difference in path metrics at 
the remerge point is given by 



A, - Ai - 2 (a n - a' n ) 2y n - (a„ - a' n ) 

n-p+1 L 



Xo 



-2 I ««b-flUx M .+ <fc 

m~n—M 



— b' m ) f„-m} 



+ I (bn-b'n) 

n=p+l 



2Zn~ (bn-b' n )XO 



-2 1 {(b m - b' m ) Xn . m - (a m - aU) $•„-„} 

m— 7i— M 



(20) 



where 



and 



Let 



A q is the metric of the correct path, 

A, is the metric of the remerging incorrect path, 

a n , b n are the channel symbols along the correct path, 

a'n, b' n , are the channel symbols along the incorrect path. 
E n = % (a n - a f n ), 

Fn = K(b n -b'n). 



(21) 
(22) 



Then, substituting (10) and (11) into (20), and after extensive simpli- 
fication (see Ref. 18 for a similar derivation), we obtain the result 



where 



Aq — A'q = D + n eq , 



D= % E n En + 2 % (E mXn _ m -F m Zn-m) 
n=p+l m=n—M 



+ 2 F n Fn + 2 I (F mXn - m + E m $n- m ) 

n=p+i m=n—M 



(23) 



(24) 
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Fig. 2 — State transition diagram for four-level QAM with four symbols of isi. 



and n eq is a zero-mean Gaussian noise random variable with variance 



o 2 = DNo/2. 



(25) 



In (24), we have assumed that the energy contained in h(t) is unity. 

At the <7th epoch, the mlse receiver must choose a survivor from 
among that path agreeing with the correct path over the most recent 
M epochs and all other incorrect paths remerging with it. An error will 
be committed if an incorrect path is chosen. Since the correct path 
may have been deleted by a prior decision, the metric comparisons 
may be between the remerging incorrect path and a partially correct 
path with metric greater than that of the overall correct path. The 
probability of committing an error at the qth epoch by selecting a 
particular one of the remerging incorrect paths of length q — p is, then, 
overbounded for a particular correct sequence by 
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P p <P{A 9 -A;<0}. (26) 

Recognizing that the true energy contained in h(t) is xo = eb, we 
now normalize the intersymbol interference coefficients by dividing 
each by xo to obtain the result 

P p <Q(J2e b D/N ), (27) 

where Q ( • ) is the complementary error function. 

Let N p be the number of incorrect paths which diverge from the 
correct path at the pth epoch and remerge, for the first time, at the 
qth epoch, and let S p j be an error event associated with one such path. 
Then the probability of committing an error at the gth epoch is 
overbounded, for a particular correct path, by 

Pe < P \ U U S pJ \ . (28) 

Applying the union bound, 

Pe<Z £p{#pj}, (29) 

p=l y=l 

where, for a particular event S p j, P{S P j} is the probability that A g - 
A^<0. 

Let fipj be the number of bit errors associated with the event S p j. 
Then, the expected ber for decisions made at the qth epoch, condi- 
tioned on a particular correct sequence, is overbounded by: 

p^ISr^W,}] • (30) 

P-1 7-1 ° K K=l 

The particular form chosen for (30) will be useful later. 

Finally, we note that the correct path is one of I q ~ p outcomes over 
the unmerged span between epochs p and q. The result (30) must, 
therefore, be averaged over these outcomes to remove the conditional 
dependence on the correct path. 

Thus, evaluation of the upper bound involves a weighted summation 
of the probabilities of all error events merging with the correct path at 
the qth node. We will now develop a technique for performing this 
summation in an orderly fashion. 

We note from (25) that for every error event, D > 0. Also, for each 
error event, D is dependent only upon the error sequences (E n , F n ), 
p < n < q. For one particular error event, the quantity D will 
assu me its minimum value D m i n . Then, using (27) and the inequality 1 
Q(Vx + v) < (y/x) e~ y/2 forx>0,y> 0, we obtain 

P B < Q ( yj^f 1 ) exp {ebDr^/No) 

'il^-[^exp{-e b Dp tJ /No}]. (31) 

p=l j=\ OK 
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Hence, evaluation of the upper bound involves computation of the set 
of Euclidean distances D p j for each correct path. The error-state 
transition matrix, to be described, provides a systematic way to find 
these distances, average over correct paths, and perform the sum. 

Referring to eq. (24), we see that the contribution to the distance 
between a correct and an incorrect path incurred between nodes / and 
j + 1, where p < y < <?, is given by 

j-i 



d - J2ejNi Ej\ Ej + 2 £ (E m Xj- m + **«&-, 

I m=j-M 



7-1 



+ 42ejNoFj | F, + 2 £ (F mXj . m - E m $j- m ) 

m—j—M 



(32) 



This contribution can be represented as the distance incurred by a 
transition from error state Ej-m, Ej-m+\> • • - , Ej-i, Fj-m, Fj-m+i, • • • , 
Fj-i to error state Ej-m+i, Ej-m +2 , Ej, Fj-mm, Fj-m**, • • • , Fj. A particular 
pair of elements (E n , F n ) from an error sequence may assume no more 
than </</(/ — 1) + 1 discrete values. For example, if a n and b n are 
binary and independent, then the pair (E n , F n ) may assume the nine 
values: (0, 0), (0, -1), (0, 1), (1, -1), (1, 0), (1, 1), (-1, -1), (-1, 0), 
(— 1, 1). Excluding the all zeros error state, J M - 1 error states remain. 
We write down a (J M — 1) X (J M — 1) error-state transition matrix 
T = {ti,i}, where the indices i and I each range over the J M — 1 integers 
labeling the error states. The element U,i represents the weighted 
distance incurred by the i -* I error-state transition: 

and 

du is the Euclidean distance incurred in permissible transi- 
tions i — » /; 

fiij is the number of bit errors incurred in the i — » I transition; 

fi,i is a factor representing the fraction of data pairs (ay, bj) 
which can produce the error (jBj, Fj) at any epoch/ 

Element ty = for forbidden transitions. 

The weighted distance incurred along any error path between epochs 
j\ and #a can be uniquely determined from either the error sequence 
between j\ — M andy'2 or the error-state transitions between j\ and 72. 
Using the latter approach, the weighted distance incurred along an 
error path between epochs j\ and j 2 is simply the product of the 
weighted transition distances tu along that path. 

As an example of the values to be assigned to the factors fa and 
Hi,i, suppose that the Zth error state at epoch j is characterized by 
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{Ej, Fj) = (1, —1), and the channel symbols a, and bj are binary and 
independent. Then, a, and bj must equal +1 and — 1, respectively, 
implying that fa = Ya, and the number of errors incurred /a,,/ = 2. 
Alternatively, if (Ej, Fj) = (1, 0), then aj must equal +1, bj can be ±1, 
fu = x h, and /*,,/ = 1. 

Since each error state can be reached from no more than J — 1 error 
states, each row in the error-state transition matrix contains no more 
than J — 1 nonzero entries, and T is quite sparse. 

The error states are numbered from 1 to J M — 1. Then, at any epoch 
j — 1, an error event may begin by a transition from the correct state 
(Ej- M = 0, • ■ • , Ej-x = 0, Fj-m = 0, • • • , Fj-i = 0) to one of the J - 1 
states (Ej-m+i = 0, • • • , Ej- X = 0, Ej, Fj- m+1 = 0, ■ ■ • , F/-, = 0, Fj). We 
introduce the J M — 1 dimensional column vector V 9 . The nth element 
of \ q is the accumulated weighted distances of all error paths spanning 
q epochs which merge with the nth error state at any node. Then, Vi 
contains J — 1 nonzero elements, corresponding to the J — 1 error 
states given above. The values assumed by these J — 1 elements are 



U = J2e b /N [Ej + F}] K »fi, (33) 

where 

Hi is the number of bit errors incurred in transitioning from the all- 
zeros error state to error state i; 

fi is the fraction of data pairs (a y , bj) capable of producing a transition 
from the all-zeros error state to error state i; 

and 

i can assume one of the J — 1 values corresponding to the indices of 
error states which can be reached from the all-zeros error state. 

Also, the vector V q is given by 

V(? = T («7-D Vi (34) 

At depth p, a remerge may occur from the J — 1 error states 
(Ej, E J+1 = 0, • • • , Ej +P -! = 0, Fj, Fj+i = 0, • • • , F j+P -! = 0). We introduce 
a second J M — 1 dimensional vector U containing only J — 1 nonzero 
elements. The nth element of U is equal to unity only if error state n 
merges with the all-zeros error state. Then the contribution to the ber 
bound at depth p from the point of divergence is given by 



Pb, p <V t - 



— (T p_1 Vi) 
Ac 



(35) 



Thus, using the error-state transition matrix, the ber bound (30) 
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can be averaged over the correct sequence and rewritten as: 






- (T^Vi) 
8k 



Q 



2ebDn^\ 

~ 1 exp (ebDmin/No) . 



An alternate form is 

Pb<U t 



No / 



[T p -'( K = 1 + e)Vi - T p ~Hk = l)Vi] 



(36) 



(limii 

Q(yJ^^\exp(e b D min /N )\. 



The bound will converge if the matrix series 

S= £ T p - 1 ( K = 1) 

converges. If (38) converges, then 

S = (I-T)-\ 
and 



(37) 



(38) 



(39) 



P fl <U 3 



(I _ T) -i Vl 



. Q( ^ebDunn/NcJexpiebDrnin/No). (40) 



A necessary and sufficient condition for the convergence of S is that 
the largest eigenvalue of T have magnitude less than unity. 

If J M — 1 is small, then the largest eigenvalue of T can actually be 
found, and if the convergence criterion is satisfied, then the ber bound 
for mlse can be found in closed form by inverting I — T and substi- 
tuting into (40). When J M — 1 is too large to conveniently admit such 
closed-form solution, then mathematical rigor must be sacrificed by 
returning to (37), and truncating the series at some point p ma x such 
that the remainder may safely be assumed to be small. In 
this event, the order of matrix T is usually quite large, and some 
concern may arise from the apparent need to repetitively multiply 
large matrices. This is not the case, however, since the term T p-1 Vi 
may be found recursively as 



T P-i Vl = T(T p - 2 V 1 ), 



(41) 



i.e., the term T p J Vi may be found by recursively multiplying the 
matrix T by a vector. 
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IV. APPLICATIONS 

As a first application, we consider mlse detection of 4<f> psk with 
overlapping rectangular pulses. Let 

h(t)=p(t)e J ", (42) 



where 



Then, 



Pit) 



-G 



VW 



T, 



0<t<T 

elsewhere . 



S(t) = Re { I (a k +jb k ) h(t - kT)e 



k-0 



<-}• 



(43) 



(44) 



where T < t is the signaling interval, and a k and 6* are independent 
and equally likely to be ±1. Suppose T = t/4, i.e., 8 bits are transmitted 
every symbol duration (t seconds). If A = 0, the four possible signal 
points for each transmitted symbol are as shown in Fig. 3a, correspond- 
ing to the familiar 4<j> psk modulation. Similarily, if A = 27t/t, then 
K(t-kT) = p{t - kT)exp[jA(t - kT) - p(t - kT)exp(-f k/2 ) . 
expij 2 ""*), and the four signal points for each symbol transmitted are 
again as shown in Fig. 3a, although the points exhibit a progressive 90- 
degree phase rotation for each successive symbol transmitted. The 
carrier is also offset to the new value to' = wo + 2tt/t. However, since 
a k and bk are independent and equally likely to be ±1, the error events 
for the waveshapes corresponding to A = and A = 2tt/t are identical, 
and both must, therefore, exhibit the same detection performance. 

For A - n/r, h(t - kT) = p(t - kT)e~ rk " e~ r '\ the carrier is offset 
to the new value to' = too + "rr/r, and the signal points alternate on 



(a) 



(b) 



Fig. 3 — Signal constellations for four-level qam. (a) 4$ psk; carrier offset A = or 
7t/t. (b) 4<£ psk; carrier offset A = tt/t. Signal constellation alternates between crosses 
and circles. 
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successive symbol transmissions between the two sets of locations 
shown in Fig. 3b. As will subsequently be shown, detectability for 
A = 7t/t is superior to that for A = 0. 

For rectangular signaling with T = t/4, it is readily established that 
M = 3, and the mlse receiver for 4$ psk contains 4 3 = 64 states. The 
error-state transition matrix needed for analysis, however, is of the 
order N = 9 3 — 1 = 728. We number the error states by means of the 
six-tuple (i£i, F\, E 2 , F 2 , E3, F3), where each element can assume 3 
values (0, ±1). The state where each element is equal to zero corre- 
sponds to a merge and is excluded from the error-state transition 
matrix. For numerical purposes, a convenient way to number the error 
states is given by 

1 + 3°(F 3 + 1) + 3 l (E 3 + 1) + 3 2 (F 2 + 1) 
+3 3 (E 2 + 1) + 3 4 (Fi + 1) + 3 5 (£i + 1), 
B=\ Bs(9»-l)/2, m 

3°(F 3 + 1) + 3 1 (£ 3 + 1) + 3 2 (F 2 + 1) 
+3 3 (E 2 + 1) + 3 4 (Fj + 1) + 3 5 (£i + 1) , 

B > (9 M - l)/2, 

where B is the state number. If the above numbering scheme for a*, bk 
binary and independent is generalized to arbitrary M, then it can 
readily be established that the error-state transition matrix possesses 
the symmetry: 

Uj = fp+1-i.p+i-y, P = 9 M - 1 . (46) 

Each row in T contains no more than nine nonzero elements, corre- 
sponding to the permissible transitions. 

For the case A = 0, Dmin was found by search to be 0.5, and for A = 
7t/t, Dmin was found to be 0.757. Bit-error rate upper bounds for the 
two cases as functions of eb/No are shown in Fig. 4. Also shown, for 
comparison, is the ideal ber for quadraphase signaling with nonover- 
lapping rectangular pulses, i.e., 2 bits transmitted per symbol duration 
T. We see that, for 10~ 6 < P B < 10" 4 , mlse performance for t = t/4, A 
= w/t is within 1.5 to 2 dB of that achieved with nonoverlapping 
pulses, although the signaling rate is four times as high. Also, over the 
same ber range, performance of signaling with A = is 1.5 to 2 dB 
poorer than with A = m/r. 

We next consider the transmission of overlapping pulses of finite 
duration with amplitude tapering to contain spectral emission. The 
pulse h(t) takes the form: 

K(t\ = | [1 + € cos (2^)/r]exp(/') , - t/2 < t < t/2, 
[0, elsewhere, 
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Fig. 4 — Bit-error rate vs €b/N for overlapping rectangular pulses, with no carrier 
offset and with optimum carrier offset. Four symbols are transmitted per pulse duration 
L. 



where < e < 1. The transmitted signal is given by (44) with h (t) as 
above, and the power spectrum for various values of e, with A = 0, are 
as shown in Fig. 5. As e increases in amplitude, the main lobe of the 
power spectrum tends to increase, but the sidelobes are reduced. 

For the particular case e = 0.45, the peak sidelobe is down 24 dB 
relative to the main lobe height. Let us define the R.F. bandwidth as 
being twice the baseband frequency at which the power spectrum dips 
permanently below -24 dB. Then, from Fig. 5, the spectral efficiency 
of the transmission (bit rate/R.F. bandwidth) is equal to (2/2.5) (t/T) 
bits/s/Hz. where, as before, T" 1 is symbol signaling rate. 

The mlse error rate bounds for this pulse appear in Fig. 6. For T = 
t/4, corresponding to M = 3, it was found that Dmin occurs at A = 77-/2t, 
and for T = t/5, corresponding to M - 4, it was found that -D min occurs 
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at A = 5tt/8t. Both cases are compared against A = 0. Results for 
T = t/4, corresponding to a transmission rate of 3.2 bits/s/Hz, show 
that for Pb = 10" 4 , optimum carrier offset is about 0.5 dB superior to 
no offset, and that eb/No must increase by about 4 dB, compared 
against nonoverlapping pulses, to maintain the error rate. However, 
for the same pulse shape, transmission of nonoverlapping pulses pro- 
vides a rate of only 0.8 bits/s/Hz. Thus, the transmission rate is higher 
by a factor of four. For T = t/5, the transmission rate is 4 bits/s/Hz, 
(a factor of five higher than for nonoverlapping pulses), and the 
required increase in eb/No is about 5.1 dB for optimum carrier offset. 
Performance without carrier offset is, again, about 0.5 dB inferior. 



s 

2 -20 



-30 



-40 



P(f) = - 



1 +£COS(27Tf/T), -T/2 < Kt/2 
0. ELSEWHERE 




l/T 2/T 

FREQUENCY 



3/T 



Fig. 5 — Spectrum of raised-cosine pulses. p(t) = 1 + £Cos(27t^/t) for —t/2 < t < t/2, 
and p(t) = elsewhere. 
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Fig. 6 — Bit-error rate for overlapping raised-cosine pulses with c = 0.45, for no carrier 
offset and optimum carrier offset. 

Let us compare these results against M-ary psk signaling with 
nonoverlapping pulses. We define the carrier-to-noise (cnr) p required 
to achieve a given ber by 



p = Re b /N , 



(48) 



where R is the transmission rate in bits/s/Hz. For nonoverlapping 
binary pulses (4<>-psk), the transmission rate is 0.8 bits/s/Hz, and to 
maintain a ber of 10~ 4 , e b /N = 8.4 dB and p = 7.4 dB. If we double 
the transmission rate to 1.6 bits/s/Hz by employing 16</> psk with 
nonoverlapping pulses, the required cnr ratio increases by 12 dB to 
p = 19.4 dB. By contrast, mlse with optimum carrier offset provides 
a transmission rate four times higher (3.2 bits/s/Hz) with p = 17 dB, 
and a transmission rate five times higher (4 bits/s/Hz) with p = 19.7 
dB. Thus, for p = 19.7 dB, quadraphase modulation with overlapping 
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pulses, optimum carrier offset, and mlse can more than double the 
capacity compared against 16</> psk with nonoverlapping pulses. 

As a final example, we give perhaps our most dramatic demonstra- 
tion of the inherent power of mlse. We consider thath(t) is the impulse 
response of a four-pole Butterworth filter whose center frequency is 
offset from the carrier frequency. At baseband, the matched-filtered 
response to h(t) has a spectrum 



H(co) = 



1 + : «-«i 

C03dB 



(49) 



where wi = olojmb is the carrier offset, and a>3dB is the cutoff frequency 
of the Butterworth filter. We arbitrarily define the R.F. bandwidth as 
twice the 3 dB cutoff frequency for no carrier offset. Transmission of 
S(t), (44), at a rate T~ l — W3dB/27r then corresponds to R = 1 bit/s/Hz. 
We consider transmission of overlapping pulses at a rate five times as 
high. The various Xn's and f n 's are found from (15) and (16) via contour 
integration. The impulse response is truncated at M = 4, and | x« | and 
| f„ | are both less than 0.02 for n > 4. 

Results are as shown in Fig. 7. The optimum carrier offset was found 
to be a = 0.625, yielding Dmin = 0.9; for a = 0, Dmm = 0.8. We see that, 
for both cases, results are close to ideal rectangular signaling with 
nonoverlapping pulses, and that the optimum carrier offset is superior 
to no offset by about 0.5 dB. To maintain a ber of 10 -4 , eb/No = 
9.5 dB. Since the transmission rate is 5 bits/s/Hz, the required cnr is 
then p = 16.5 dB. At this cnr, Shannon's channel capacity C/W = 
log2(l + p) is 5.5 bits/s/Hz. Thus, for P e = 10 -4 , mlse performance 
with the simple Butterworth signaling chosen is quite close to the 
Shannon limit. Even without carrier offset, the required cnr is about 
17 dB, for which the Shannon limit is 5.7 bits/s/Hz. 

V. CONCLUSION 

Maximum-likelihood-sequence estimation has been shown to be a 
powerful technique to permit high rates of data transmission with 
straightforward channel signaling. In this treatment, channel coding 
techniques were not used, and Quadrature Amplitude Modulation was 
employed. Implementation of mlse by means of the Viterbi Algorithm 
requires a detector with I M states, where / is the number of levels 
which may be assumed by each channel symbol and M is the extent, 
in symbol intervals, of intersymbol interference. Complex signaling 
waveforms, which are generated by means of carrier offset modulation, 
can be accommodated and may improve detection efficiency; mlse 
equalizes both the in-phase and quadrature components of the inter- 
symbol interference. 
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Fig. 7— Bit-error rate vs e b /N for four-pole Butterworth transmit filtering for no 
carrier offset and optimum carrier offset. 

An error-state transition analysis was used to systematically com- 
pute a ber bound for mlse. Applying this bound, it was shown by 
example that, with mlse, transmission rates approaching the Shannon 
limit may be possible without channel coding. 

The power of mlse detection makes this technique a candidate for 
radio systems constrained in bandwidth. Since the mlse approach 
admits complex waveshapes, channels admitting frequency-selective 
fading can readily be accommodated, provided that the slowly varying 
channel frequency characteristics are monitored to update the detec- 
tion constants continuously. The real limitation on the application of 
mlse involves the speed at which the detector can operate, and this 
places a constraint on the symbol rate of the radio channel. For rates 
under 30 Megabaud, mlse may be quite feasible; implementation at 
real-time rates of 300 Megabaud, such as may apply to satellite 
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channels, is beyond the current state-of-the-art. However, if the sat- 
ellite system employs time-division multiple access (tdma), then the 
low-duty cycle for any ground station may permit mlse implementa- 
tion in nonreal time between tdma burst arrivals. For duty cycles 
lower than 10 percent, this would appear to be feasible today. At 
higher duty cycles, nonreal time detection with parallel processors is 
another possibility. 
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